A lack of corpora has so far limited advances in integrating human gaze data as a supervisory signal in neural attention mechanisms for natural language processing (NLP). We propose a novel hybrid text saliency model (TSM) that, for the first time, combines a cognitive model of reading with explicit human gaze supervision in a single machine learning framework. On four different corpora we demonstrate that our hybrid TSM duration predictions are highly correlated with human gaze ground truth. We further propose a novel joint modeling approach to integrate TSM predictions into the attention layer of a network designed for a specific upstream NLP task without the need for any task-specific human gaze data. We demonstrate that our joint model outperforms the state of the art in paraphrase generation on the Quora Question Pairs corpus by more than 10% in BLEU-4 and achieves state of the art performance for sentence compression on the challenging Google Sentence Compression corpus. As such, our work introduces a practical approach for bridging between data-driven and cognitive models and demonstrates a new way to integrate human gaze-guided neural attention into NLP tasks.

To gain further insight into the comparison between our model and the current state of the art in sentence compression, we show results of our method and ablations in relation to ablations of the method by Zhao et al. (see Table 1). In their work, the authors added a "syntax-based Also shown is the number of model parameters. We show that our model, without additional syntactic information as was used in previous methods, still obtains SOT A performance. Figure 1: Additional paraphrase generation attention maps from our ablation study, for both sub-networks (TSM predictions and upstream task attention) in our joint architecture. TSM fixation predictions (left in blue) over epochs (last epoch is our converged models). However, we assume they do not play a role in performance between these two conditions, as these performance differences are not statistically significant.

Weaknesses: There are multiple issues with the claims and evaluations presented in the paper. In particular, as a reader, I am not convinced that reported gains are due to exploiting gaze information. An improvement over SOTA?: For paraphrasing task, the paper claims Patro et al. (2018) as SOTA which is an outdated baseline. Given that "No Fixation" method gives 27.81 BLEU-4 score with 69M params, I doubt that the proposed model's 28.82 BLEU-4 score with 79M is truly better than Patro et al. (2018)'s model. Ideally, authors should report the performance of baseline models using the same number of parameters.

Overall, the reviewers appreciated this paper and thought it was an interesting contribution to the literature. Because of this I am recommending that the paper be accepted. However, there was one major request from the Reviewer 3 (and me) that the authors appropriately frame their discussion of previous work, specifically [3]. Quoting reviewer 3's discussion directly: "The authors claim to be first to directly supervise attention; which has been done before with token-level annotation, and is exactly what [3] does with gaze. This false claim of novelty is problematic, but also unnecessary, since this is a great paper that already makes decent contributions, eg smart pretraining."

A lack of corpora has so far limited advances in integrating human gaze data as a supervisory signal in neural attention mechanisms for natural language processing (NLP). We propose a novel hybrid text saliency model (TSM) that, for the first time, combines a cognitive model of reading with explicit human gaze supervision in a single machine learning framework. On four different corpora we demonstrate that our hybrid TSM duration predictions are highly correlated with human gaze ground truth. We further propose a novel joint modeling approach to integrate TSM predictions into the attention layer of a network designed for a specific upstream NLP task without the need for any task-specific human gaze data. We demonstrate that our joint model outperforms the state of the art in paraphrase generation on the Quora Question Pairs corpus by more than 10% in BLEU-4 and achieves state of the art performance for sentence compression on the challenging Google Sentence Compression corpus.